Method and system for developing and administering subject-appropriate implicit-association tests

ABSTRACT

The current application is directed to developing and administering subject-appropriate ITAs. The various to developing and administering subject-appropriate ITAs evaluate many aspects of ITA content, content presentation, and administration with regard to particular categories of test subjects in order to develop subject-appropriate ITAs (“SAITAs”) and to administer SAITAs appropriately to the categories of test subjects for which the tests are developed. Aspects of SAITA presentation and administration that are evaluated include input devices, presentation formats, presentation language, presentation media, colors used in presentation of tutorials and stimuli, words, pictures, and symbols used in a presentation of tutorials and stimuli, the pace and length of various portions of SAITAs, human vs. computer administration of SAITAs, presentation of feedback to test subjects during SAITAs, ITAs and many other aspects of ITA content, content presentation, and administration. SAITAs employ systematic methods for both development and administration of SAITAs.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation-in-part of application Ser. No.12/150,947, filed Apr. 30, 2008 now abandoned.

STATEMENT OF GOVERNMENT INTEREST

This invention has been made with Government support under Contract No.SBE-0354453, awarded by the National Science Foundation. The governmenthas certain rights in the invention.

TECHNICAL FIELD

The present invention is related to psychological testing and, inparticular, to a method and system for developing subject-appropriateimplicit tests of associations and for administering subject-appropriateimplicit tests of associations in subject-appropriate ways.

BACKGROUND OF THE INVENTION

The implicit-association tests, first described in a 1998 publication,was developed to test the strength of subjects' mental associationsbetween concepts and ideas based on the test subjects' performance onrelatively simple, computer-proctored tests during which subjectscategorize various computer-presented stimuli into different categoriesby manual input via a keyboard or other input device.Implicit-association tests (“IATs”) are described, in greater detail, ina following section. Please note that the term “subject” refers to aperson to whom an IAT or other test is administered. Please also notethat, although not currently used in the literature, a perhaps bettername for the implicit association test is “implicit test ofassociations.” The test is implicit, in the sense that the test does notquery test subjects for explicitly-stated responses. However, it isoften assumed that the adjective “implicit” modifies the term“association,” which is incorrect. Therefore, in the following, the name“implicit test of associations” (“ITA”) is used, in preference to thecommonly used “implicit association test.”

While ITAs have proved effective in evaluating strengths of associationbetween various concepts and categories held by reasonablywell-educated, literate, non-disabled adults, the standard ITA formathas proved difficult or impossible to administer to various othercategories of test subjects, including preschool children, illiterateadults, people with various types of disabilities, and others.Furthermore, ITA development has been largely empirical, generallywithout a systematic approach to evaluating particular ITA suitabilityfor various categories of subjects. ITA administrators and developershave recognized the need for methods and systems for more effectiveadministration of ITAs to, and development of ITAs for, particularcategories of subjects.

SUMMARY OF THE INVENTION

The current application is directed to developing and administeringsubject-appropriate ITAs. Various methods disclosed in the currentapplication evaluate many aspects of ITA content, content presentation,and administration with regard to particular categories of test subjectsin order to develop subject-appropriate ITAs (“SAITAs”) and toadminister SAITAs appropriately to the categories of test subjects forwhich the tests are developed. Exemplary categories of test subjects mayinclude adults with a particular class of disabilities, preschoolchildren, illiterate adults, elderly adults, and emigrants lackingexposure to computers and to the dominant culture of the society inwhich tests are developed and administered. Aspects of SAITApresentation and administration that are evaluated include inputdevices, presentation formats, presentation language, presentationmedia, colors used in presentation of tutorials and stimuli, words,pictures, and symbols used in a presentation of tutorials and stimuli,the pace and length of various portions of SAITAs, human vs. computeradministration of SAITAs, presentation of feedback to test subjectsduring SAITAs, administration of Likert scales as a method ofverification of correctly administered SAITAs, continuous monitoring ofthe test-subject's responses, attitudes toward the stimulus/responseinteraction, attentiveness, and other characteristics to ensure thatmeaningful responses are being measured during the SAITA, additionalcontinuously applied internal consistency checks and feedback loops, andmany other aspects of ITA content, content presentation, andadministration. Testing approaches to which the current application isdirected employ systematic methods for both development andadministration of SAITAs.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-F illustrate the general approach of the ITA.

FIGS. 2A-E illustrates processing of elapsed times of response, orlatencies, measured during administration of the exemplary ITA describedwith reference to FIGS. 1A-F.

FIGS. 3A-B illustrate the principal aspects of ITA administration thatneed to be tailored for specific categories of subjects in order todevelop and administer SAITAs.

FIG. 4 illustrates various innovations regarding input devices as wellas stimulus and information presentation.

FIG. 5 illustrates a visually displayed screen that has been developedto test strengths of association tested by the standard ITA in FIGS. 1A-F.

FIG. 6 illustrates a method of audio-stimulus presentation in an SAITAdirected to preschool children.

FIG. 7 illustrates presentation of a visual stimulus in an SAITAdirected to preschool children.

FIG. 8 illustrates an additional aspect of visual presentation ofstimuli in an SAITA directed to preschool children.

FIGS. 9A-B illustrate an additional aspect of the SAITA directed topreschool children.

FIG. 10 illustrates an additional aspect of the SAITA directed topreschool children.

FIG. 11 illustrates the standard ITA as a control-flow diagram.

FIGS. 12-17 illustrate a general SAITA using the control-flow-diagramillustration method employed in FIG. 11 to illustrate the standard ITA.

FIG. 18 illustrates various types of information that may be employed tofacilitate development of SAITAs.

FIG. 19 illustrates, in control-flow fashion, the design and methodologyfor designing SAITAs.

FIGS. 20A-F illustrate an example of a subject-appropriate Likert-typetesting method that solicits explicit responses from subjects related togender and math skills.

FIG. 21 illustrates a general-purpose computer architecture suitable forexecution of IAT, SAITA, and other types of tests.

DETAILED DESCRIPTION OF THE INVENTION

The current application is directed to development and administration ofsubject-appropriate implicit test of associations (“SAITAs”) based onpreviously developed implicit tests of associations (“ITAs”). Tofacilitate description of the SAITAs disclosed in the currentapplication, standard ITAs are first described.

FIGS. 1A-F illustrate the general approach of the ITA. The ITA isadministered via a computer, generally a personal computer orworkstation. The stimuli that elicit test-subject responses aregenerally provided visually, on a computer display 103. Subjects respondto stimuli by depressing particular keys of a keyboard 104. Duringinitial phases of ITA administration, a subject is instructed to depressa first key 108 when pictures or other visual stimuli related to a firstpair of concepts 106 are presented to the subject and to depress asecond key 112 when pictures or other visual stimuli related to a secondpair 110 of concepts are presented to the subject. During these initialphases, the subject is provided with practice stimulus/responseinteractions to facilitate learning the concept-pair/response-keycombinations. The first concept in each pair of concepts, 114 and 116,constitute a first conceptual dimension, or target concepts, and thesecond concept in each pair of concepts, 118 and 120, constitute asecond conceptual dimension, or attribute concepts. In the example shownin FIG. 1A, the target concepts relate to objects, and include the twodifferent types of objects: {flowers, insects}. The attribute concepts,in the example shown in FIG. 1A, include the two feelings or emotions:{pleasant, unpleasant}.

During the testing phase of an ITA, the test subject is instructed todepress the appropriate key in response to presentation of a stimulus,generally a picture or phrase that clearly belongs to one of the fourconcepts within the two pairs of concepts. In the case of the example ofFIG. 1A, each presented stimulus is clearly associated with, or belongsto, one of the four concepts: {flowers, insects, pleasant, unpleasant}.FIG. 1B illustrates an exemplary stimulus presentation. In FIG. 1B, apicture of ice cream and candy 126 is displayed on the computer displaydevice 102. At the instant at which the picture is displayed, acomputational timing device 128 is initialized. As shown in FIG. 1C, theuser, in response to viewing the stimulus image, depresses 130 the key108 appropriate for inputting a response to visual display a concept ofthe concept pair {flowers, pleasant} 106, since candy and ice creamclearly associate with the concept “pleasant,” and do not associate withthe other concepts “flowers,” “insects,” and “unpleasant,” at least tothose without potentially fatal allergies to ice cream and candy. At theinstant when the key is depressed by the subject, the elapsed timebetween display of the stimulus and the user response is measured viathe computational timing device 128. The computational timing device isgenerally based on a hardware system clock and various relatedoperating-system functions.

In general, an ITA test comprises two blocks, each block in turncomprising a fixed number of stimulus-presentation/test-subject-responsepairs, such as the stimulus presentation and response illustrated inFIGS. 1B-C. During the first block, the concept-pair/response-keycombinations remain constant. Then, the relationship between the keys tobe depressed and the concepts or categories presented to the subject ischanged. In a tutorial administered during the second block, the subjectis instructed, as shown in FIG. 1D, to depress a first key 108 whenpictures or other visual stimuli related to the concept pair {flowers,unpleasant} 134 are presented to the subject and to depress a second key112 when pictures or other visual stimuli related to the concept pair{insects, pleasant} 136 are presented to the subject. The subject isagain provided with practice stimulus/response interactions tofacilitate learning the new concept-pair/response-key combinations. Asshown in FIGS. 1E-F, the stimuli, such as picture 126, are againpresented, during the second block, and the elapsed time betweenstimulus presentation and subject response are again measured. Ingeneral, the target concepts, or concepts of the first conceptualdimension, are reversed, in the second block, although it is possible toreverse the attribute concepts, or the concepts of the second conceptualdimension, in alternative examples.

Often, the developer of an ITA begins with a hypothesis. For example,with regard to the ITA described with reference to FIGS. 1A-F, thedeveloper may hypothesize that people generally have a preference forflowers over insects (i.e. associate flowers with pleasant emotions andinsects with unpleasant emotions). Therefore, during the first block,the concept-pair/response-key combinations reflect the hypothesizedassociation between flowers and pleasant emotions and between insectsand unpleasant emotions, and the first block is therefore referred to asthe “congruent block.” By contrast, the second block, during which thekeys are associated with concepts that are hypothesized to be clashing,or discordant, is referred to as the “incongruent block.” While theconcepts or categories paired in the congruent block may be generallyhypothesized to reflect natural associations held by one or more classesof subjects and the concepts or categories paired in the incongruentblock may be generally hypothesized to run counter to associationscommonly held by one or more classes of subjects, the responses of anygiven subject during an ITA may indicate that the subject stronglyassociates the concepts or categories paired in the congruent block,weakly associates the concepts or categories paired in the congruentblock, exhibits an equally strong association between the concepts orcategories paired in the congruent block, weakly associates the conceptsor categories paired in the incongruent block, or strongly associatesthe concepts or categories paired in the incongruent block. In certainITAs, the test developer may not have a prior assumptions of thestrength of association between the different concepts or categories forwhich strength of association is tested, in which cases the phrases“first block” and “second block” may be preferred to the phrases“congruent block” and “incongruent block.” FIGS. 2A-E illustratesprocessing of elapsed times of response, or latencies, measured duringadministration of the exemplary ITA described with reference to FIGS.1A-F. A first table 202 includes the elapsed times of responses to eachof a series of pictures displayed in a practice session within thecongruent block of the test, a second table 204 includes the elapsedtimes of responses to each of a series of pictures displayed in the testportion of the congruent block, a third table 206 includes the elapsedtimes of responses to each of a series of pictures displayed in apractice session within the incongruent block of the test, and a fourthtable 208 includes the elapsed times for responses to pictures presentedas stimuli in the test portion of the in congruent block. Note that thesame pictures are presented in all four portions of the ITA, althoughthe order of presentation may vary. Each entry in each table includes anindication of the stimulus to which the latency applies, as well as anindication of whether or not the response was correct. In general, asubject is required to provide a correct answer to each displayedstimulus. When the response is incorrect, a visual indication isdisplayed, and the subject is required to input the correct response. Ingeneral, latencies for initially incorrect responses are thereforegreater in value than latencies for correct responses, due to the needto input a second, correct response following an initially incorrectresponse.

There are a variety of ways to compute meaningful scores. The validityof different scoring methods can be tested by comparing scores derivedfrom ITA response latencies to subjects' preferences for, oridentification with, concepts and categories measured by alternative,self-expression-based testing, such as Likert-type testing. One approachto scoring, for a two-block ITA test, each block of which includes apractice strength-of-association-test session (“P”) and astrength-of-association test (“T”) is next provided, with reference toFIGS. 2B-E.

First, four sets of data P1, T1, P2, and T2 are defined as the latenciesmeasured for the above-mentioned practice sessions and test, as shown inFIG. 2B and expressed below:P1={P_(1,1), P_(1,2), . . . , P_(1,n)}=

-   -   n measured latencies for practice strength-of-association-test        session in first block, each of which is less than a cutoff        latency value        T1={T_(1,1), T_(1,2), . . . , T_(1,m)}=    -   m measured latencies for strength-of-association test session in        first block, each of which is less than a cutoff latency value        P2={P_(2,1), P_(2,2), . . . , P_(2,q)}=    -   q measured latencies for practice strength-of-association-test        session in second block, each of which is less than a cutoff        latency value        T2={T_(2,1), T_(2,2), . . . , T_(2,r)}=    -   r measured latencies for strength-of-association test session in        second block, each of which is less than a cutoff latency value

In other words, entries of the original table with latency valuesgreater than a cutoff value, in the present case 10 seconds, arediscarded to produce the four sets of data P1 (210 in FIG. 2B), T1 (212in FIG. 2B), P2 (214 in FIG. 2B), and T2 (216 in FIG. 2B).

Next, a test is made to ensure that, after the elimination of theextreme values, there are sufficient remaining values to allow for ameaningful test score to be calculated:

${if}\mspace{14mu}\begin{pmatrix}{\left( {n < {{practice\_ threshold}\mspace{14mu}{AND}\mspace{14mu} m} < {threshold}} \right)\mspace{14mu}{OR}} \\\left( {q < {{practice\_ threshold}\mspace{14mu}{AND}\mspace{14mu} r} < {threshold}} \right)\end{pmatrix}\mspace{14mu}{test}\mspace{14mu}{invalid}$Then, the number of entries in the four data sets P1, T1, P2, and T2with latency values below some low-value threshold are determined, and,if the ratio of the number of such extreme-valued entries to totalentries exceeds a threshold, in the present case, 0.1, the test isdeemed invalid, since a large number of extremely short response timesindicates that the subject was not properly evaluating displayed stimuliprior to inputting responses to the stimuli:

     P 1e = {P_(1, i) : P_(1, i) < low_cutoff_latency}     T 1e = {T_(1, i) : T_(1, i) < low_cutoff_latency}     P 2e = {P_(2, i): P_(2, i) < low_cutoff_latency}     T 2e = {T_(2, i) : T_(2, i) < low_cutoff_latency}numExtreme = sizeof(P 1e) + sizeof(P 1e) + sizeof(P 1e) + sizeof(P 1e)$\mspace{79mu}{{if}\mspace{14mu}\left( {\frac{numExtreme}{n + m + q + r} > 0.1} \right)\mspace{14mu}{test}\mspace{14mu}{invalid}}$Next, average latency for correct responses for each of the four datasets is computed, as indicated below, with the average latency valuesfor the example ITA results of FIGS. 2A-E shown in FIG. 2C:

P 1c = {P_(1, i) : answer(P_(1, i)) =  = correct}T 1c = {T_(1, i) : answer(T_(1, i)) =  = correct}P 2c = {P_(2, i) : answer(P_(2, i)) =  = correct}T 2c = {T_(2, i) : answer(T_(2, i)) =  = correct}${{avg}\left( {P\; 1c} \right)} = \frac{\sum\limits_{i = 1}^{{sizeof}{({P\; 1c})}}{P\; 1c_{i}}}{{sizeof}\left( {P\; 1c} \right)}$${{avg}\left( {T\; 1c} \right)} = \frac{\sum\limits_{i = 1}^{{sizeof}{({T\; 1c})}}{T\; 1c_{i}}}{{sizeof}\left( {T\; 1c} \right)}$${{avg}({P2c})} = \frac{\sum\limits_{i = 1}^{{sizeof}{({P2c})}}{P\; 2c_{i}}}{{sizeof}\left( {P\; 2c} \right)}$${{avg}\left( {T\; 2c} \right)} = \frac{\sum\limits_{i = 1}^{{sizeof}{({T\; 2c})}}{T\; 2c_{i}}}{{sizeof}\left( {T\; 2c} \right)}$Next, standard deviations σ_(p) and σ₁for all latencies, whether or notcorresponding to correct answers, are computed for the combined data setP1{circumflex over (0)}P2 and the combined data set P1{circumflex over(0)}P2, with the standard deviations computed for the example ITAresults of FIGS. 2A-E shown in FIG. 2C:

${avg}_{p} = \frac{\left( {\sum\limits_{i = 1}^{{sizeof}{({P\; 1})}}{P\; 1_{i}}} \right) + \left( {\sum\limits_{i = 1}^{{sizeof}{({P\; 2})}}{P\; 2_{i}}} \right)}{{{sizeof}\left( {P\; 1} \right)} + {{sizeof}\left( {P\; 2} \right)}}$${avg}_{t} = \frac{\left( {\sum\limits_{i = 1}^{{sizeof}{({T\; 1})}}{T\; 1_{i}}} \right) + \left( {\sum\limits_{i = 1}^{{sizeof}{({T\; 2})}}{T\; 2_{i}}} \right)}{{{sizeof}\left( {T\; 1} \right)} + {{sizeof}\left( {T\; 2} \right)}}$$\sigma_{p} = \sqrt{\frac{\left( {\sum\limits_{i = 1}^{{sizeof}{({P\; 1})}}\left( {{avg}_{p} - {P\; 1_{i}}} \right)^{2}} \right) + \left( {\sum\limits_{i = 1}^{{sizeof}{({P\; 2})}}\left( {{avg}_{p} - {P\; 2_{i}}} \right)^{2}} \right)}{{{sizeof}\left( {P\; 1} \right)} + {{sizeof}\left( {P\; 2} \right)}}}$$\sigma_{t} = \sqrt{\frac{\left( {\sum\limits_{i = 1}^{{sizeof}{({T\; 1})}}\left( {{avg}_{t} - {T\; 1_{i}}} \right)^{2}} \right) + \left( {\sum\limits_{i = 1}^{{sizeof}{({T\; 2})}}\left( {{avg}_{t} - {T\; 2_{i}}} \right)^{2}} \right)}{{{sizeof}\left( {T\; 1} \right)} + {{sizeof}\left( {T\; 2} \right)}}}$Next, final data sets P1 f, T1 f, P2 f, and T2 f, as shown in FIG. 2D,are computed from data sets P1, T1, P2, and T2 by substituting, forlatencies below a low-threshold value, a substitute computed latencyequal to a penalty value added to an average latency for correctanswers, and average latency values for each of the final data sets arecomputed, with the average latency values computed for the example ITAresults of FIGS. 2A-E shown in FIG. 2E:

${f\left( X_{i} \right)} = \begin{Bmatrix}{{{{when}\mspace{14mu} P_{1,i}} < {{low\_ cutoff}{\_ latency}}},{{{{avg}({Xc})} + {penalty}};}} \\{{otherwise},X_{i}}\end{Bmatrix}$ P 1 f = {f(P_(1, 1)), f(P_(1, 2)), …  , f(P_(1, n))}T 1 f = {f(T_(1, 1)), f(T_(1, 2)), …  , f(T_(1, n))}P 2 f = {f(P_(2, 1)), f(P_(2, 2)), …  , f(P_(2, n))}T 2 f = {f(T_(2, 1)), f(T_(2, 2)), …  , f(T_(2, n))}${{avg}\left( {P\; 1f} \right)} = \frac{\sum\limits_{i = 1}^{{sizeof}{({P\; 1f})}}{P\; 1f_{i}}}{{sizeof}\left( {P\; 1f} \right)}$${{avg}\left( {T\; 1f} \right)} = \frac{\sum\limits_{i = 1}^{{sizeof}{({T\; 1f})}}{T\; 1f_{i}}}{{sizeof}\left( {T\; 1f} \right)}$${{avg}\left( {P\; 2f} \right)} = \frac{\sum\limits_{i = 1}^{{sizeof}{({P\; 2f})}}{P\; 2f_{i}}}{{sizeof}\left( {P\; 2f} \right)}$Finally, a test score

${{avg}\left( {T\; 2f} \right)} = \frac{\sum\limits_{i = 1}^{{sizeof}{({T\; 2f})}}{T\; 2f_{i}}}{{sizeof}\left( {T\; 2f} \right)}$is computed as follows, with the test score computed for the example ITAresults of FIGS. 2A-E shown in FIG. 2E:

${score} = \frac{\frac{{P\; 2f} - {P\; 1f}}{\sigma_{p}} + \frac{{T\; 2f} - {T\; 1f}}{\sigma_{t}}}{2}$

When the computed test score is “0,” or close to zero, there appear tobe equally strong associations between the paired concepts or categoriesused in both blocks of the ITA. When the computed score is positive, andabove a threshold value, then the subject appears to more stronglyassociate the paired concepts or categories used in the first block thanthe paired concepts or categories used in the second block. When thecomputed score is negative, with an absolute value above the thresholdvalue, then the subject appears to more strongly associate the pairedconcepts or categories used in the second block than the paired conceptsor categories used in the first block of the ITA.

ITAs have many significant advantages over more traditional types ofpsychological tests that require subjects to provide informationalresponses to questions. In many cases, a subject may intentionally orinadvertently respond insincerely, to particular types of questions, asa result of conscious or unconscious feelings, prejudices, and a senseof the test-taker's expectations. For example, a standard psychologicaltest in which the test administrator questions the subject with regardto the subject's disposition towards members of different races may notelicit an honest expression of the subject's honest feelings andemotions, since the test is directed to a very controversial topic. Asanother example, a subject may assume that the test administrator isseeking certain types of responses, and may respond according to thatassumption, rather than to the subject's true feelings with regard tothe topic of the test. As a third example, the subject may indeed harboran unconscious bias or prejudice, but answer explicit questions withregard to the subject's disposition towards members of different racesbased on the subject's conscious attitudes related to race, such as aconscious affinity to an unprejudiced, race-indifferent disposition. TheITA, by contrast, measures response time for the test subject's responseto presented stimuli. When the responses require choosing input keysassociated with matched, or concordant concepts, subjects generallyrequire less time to respond to displayed stimuli than when the inputkeys are associated with clashing, or discordant concepts. Because thesubject is encouraged to respond quickly and mechanically to thestimuli, response times tend to be far less influenced by subjects'expectations, assumptions, prejudices, and other factors that may leadto less useful results obtained in standard psychological tests.

Problems with the standard ITA were first noticed during development ofan ITA-based approach to evaluating various concept associations inpreschool children. The standard ITA was completely unsuitable. Thestandard ITA relies on subjects being literate and having sufficientfamiliarity with computers to be able to provide responses throughstandard keyboard keys, and to respond appropriately to variousinstructional screens and feedback. Therefore, it was necessary tore-evaluate the ITA and to develop an SAITA for preschool children. Thisundertaking revealed many of the aspects of generalized SAITAdevelopment and administration. For example, because ITAs measure theelapsed times for manual responses, rather than evaluate informativeresponses to questions, subtle effects and interferences related tostimulus presentation, ITA dynamics, particular words, colors, images,sounds, and other features of the ITA format, ITA structure, and otheraspects of the ITA unrelated to the associations being evaluated maycontribute to observed elapsed response times and lead to inaccurateconclusions about the strengths of associations for which the ITA isdesigned to evaluate.

FIGS. 3A-B illustrate the principal aspects of ITA administration thatneed to be tailored for specific categories of subjects in order todevelop and administer SAITAs. One set of aspects relates to the inputdevice 302 and various visual presentations provided on the displaydevice 304 during test administration. Another aspect, illustrated inFIG. 3B, relates to the general flow of an ITA. The ITA begins withseveral introductory screens, or displays 306, which are then followedby two or more blocks, discussed above. During each block, a number ofscreens or displays 308 are devoted to instructing the subject withregard to the target-concept/response-key combinations used in theblock, a number of screens or displays 310 are devoted to instructingthe subject with regard to the attribute-concept/response-keycombinations used in the block, a practice stimulus/response sub-blockis then administered via a third set of screens or displays 312, and,finally, a test sub-block is administered in a fourth set of screens ordisplays 314. As indicated by arrow 316 in FIG. 3B, instructionregarding the concept-pair/response-key combinations, a practicesub-block, and a test sub-block are repeated for each block of an ITAtest. The flow aspect of an ITA includes a number of parameters,including the number and content of the sub-blocks within each block,such as the instruction regarding the concept-pair/response-keycombinations, the number of stimulus/response interactions in thepractice sub-block and in the test sub-block, the presentation of thestimuli in each stimulus/response interaction in the practice sub-blockand test sub-block, the type of instruction regarding theconcept-pair/response-key combinations, the number of stimulus/responseinteractions administered during instruction regarding theconcept-pair/response-key combinations, and the nature and content ofthe introduction.

By way of example, many of the innovations applied to developing andadministering an SAITA directed to preschool children are nextdescribed. These innovations illustrate the general principals of SAITAdevelopment and administration. FIG. 4 illustrates various innovationsregarding input devices as well as stimulus and informationpresentation. As shown in FIG. 4, for preschool children, the standardcomputer keyboard is modified or replaced by a subject-appropriate inputdevice 402. The subject-appropriate input device includes only two, verylarge keys or pads 404 and 406. The two keys, or pads, 404 and 406 havetwo different colors. As discussed further, below, the colors are chosento be neutral, without introducing unnecessary conceptual dimensions tothe SAITA. As one example, an SAITA directed to preschool children maybe developed to test various gender-related hypotheses. Were the twoinput keys 404 and 406 colored blue and pink, the standard, socialassociation of pink with girls and blue with boys would almost certainlyintroduce an unintended, third conceptual dimension to the SAITA.

Although any color may have unintended associations, for a particularSAITA, secondary colors orange, green, and purple tend to be generallymost neutral for preschool children. For other categories of subjects,alternative types of input devices may be most effective. For peoplelacking use of arms, due to disabilities of birth defects, spokenresponses detected by a microphone or directional orientation of thevisual axis, detected by a video camera, may be appropriate. FIG. 4includes depictions of audio speakers 408 and 410, in addition to thedisplay device 412. The standard ITA relies purely on textual andgraphical visual display of both tutorial information and thepresentation of stimuli during stimulus/response interactions. However,modern personal computers and workstations are capable of playing musicand sounds, and audio presentation proves to be useful, particularly fornon-literate subjects.

The standard ITA relies on textual concept pairings duringkey-association instruction and tutorials. In the standard ITA, thewords “flowers” and “pleasant” might be shown together, on the leftportion of a screen, and the words “insects” and “unpleasant” might beshown on the right-hand portion of the screen, indicating that theleft-hand key is associated with the concepts “flowers” and “pleasant,”while the right-hand key is associated with “insects” and “unpleasant.”This concept-pair/response-key combination presentation relies on asubject being able to read the words, and also relies on the subjectbeing able to make mental associations between the display-screenpositions of the word pairs representing concept pairs and thedesignated input keys on the keyboard. For preschool children, seeingand understanding the concept pairings and mentally establishing theconcept-pair/response-key combinations by positions of displayed text issimply too formidable a task.

In an SAITA directed to preschool children, conveyance of theconcept-pair/response-key combinations includes using colors andpictures, and matching input-key colors. FIG. 5 illustrates a visuallydisplayed screen that has been developed to test strengths ofassociation tested by the standard ITA in FIGS. 1A-F. Picturesrepresenting flowers 502-507 are shown as a collage of all exemplarsused to represent the category of flowers, above a colored band 520having the same color as the left-hand input key (404 in FIG. 4).Similarly, pictures depicting insects 510-515 are shown as a collage ofall exemplars used to represent the category insects above a secondcolored band 522 having the same color as the right-hand input key (406in FIG. 4). Thus, the target-concept/response-key combinations arenaturally and graphically presented in the visual display by matching ofthe input-key colors to colored bands 520 and 522 and the picturecollages associated with the colored bands. Thus, in an SAITA, aspecially tailored input device can be combined with matchingvisual-display features to facilitate a non-verbal imparting of theconcept/response-key combinations in at least one conceptual dimension.Similar techniques may be used for the second conceptual dimension. Forexample, in the visual display shown visual display shown in FIG. 5, acollage of pictures depicting the affective dimension “pleasant” may bedisplay below the colored bar 520, and a collage of picturesrepresenting the emotion “unpleasant” may be displayed below the coloredbar 522, so that the concept-pair/response-key combinations aredisplayed visually. Alternatively, one conceptual dimension may bevisual, as in FIG. 5, and the second conceptual dimension may beprovided by audio information presented during instruction regarding theconcept-pair/response-key combinations. In this case, the subject may betrained by audio stimulus to depress the left-hand key when pleasantwords are spoken and the right-hand key when unpleasant words arespoken. Various combinations of visual and audio presentation may beappropriate for different categories of subjects. For preschoolchildren, it has been found that representing concept/response-keycombinations related to a first conceptual dimension by visual means,and representing concept /response-key combinations related to a secondconceptual dimension by audio means, may be most effective in conveyingto the subject the concept-pair/response-key combinations that are to beemployed by the subject during each block.

FIG. 6 illustrates a method of audio-stimulus presentation in an SAITAdirected to preschool children. As shown in FIG. 6, when an audiostimulus is provided, such as the spoken word “yucky,” the word orphrase is also displayed visually on the display device 602. Visualdisplay of the word or phrase indicates to the subject that a stimulushas been provided to which the subject needs to respond. In addition,even subjects who have not learned to read may recognize the forms ofwords or phrases, and thus visual display of the words or phrases mayfacilitate subjects' understanding of the responses requested duringadministration of the ITA. In addition to displaying the word or phrase,the word or phrase is generally spoken in a way that reinforces themeaning of the word or phrase. For example, using the test discussedwith reference to FIGS. 1A-F, unpleasant words and phrases, such as“yucky,” may be enunciated dramatically in negative tones, whilepleasant words and phrases, such as “ice cream,” may be enunciated inpositive, happy tones. Achieving the needed enunciations requirescareful preparation of both speakers and recording engineers who recordthem. It is not the point of an SAITA to challenge subjects with regardto understanding presented stimulus or assigning stimuli to categories.Inferences based on response latencies are most accurate when thestimuli are readily received and understood by subjects. The criticallatency is the latency involved in deciding which of the two input keysto depress once the presented stimulus is understood and mentallyassigned to the appropriate concept.

FIG. 7 illustrates presentation of a visual stimulus in an SAITAdirected to preschool children. When a visual stimulus 702 is presented,the SAITA simultaneously provides an audio tone, such as a beep ormusical note, to reinforce to the subject the need to respond bydepressing an input key. Thus, there is symmetry between audiopresentation of stimuli and visual presentation of stimuli. In bothcases, both a sound and a visual stimulus are simultaneously presentedto the subject. In the SAITA, the pictures in the collages above thehorizontal bars 520 and 522 are the same pictures used as visual stimuliduring the SAITA blocks. This further facilitates identification, by thesubject, of the displayed visual stimulus and concept to which thevisual stimulus belongs.

FIG. 8 illustrates an additional aspect of visual presentation ofstimuli in an SAITA directed to preschool children. As discussed above,subjects are encouraged to rapidly respond to stimuli. The more rapidthe response, the greater the probability that differences in responsetimes reflect an underlying association of concepts at an unconsciouslevel. As a test subject rapidly responds to presented stimuli, the testsubject may be confused as to whether depression of a key responds tothe currently displayed stimulus, or instead elicits a next stimulus. Inone example SAITA, as shown in FIG. 8, when the subject responds to acurrently presented stimulus 802, the visual representation of thestimulus diminishes in size and moves off to one side of the visualdisplay 804, to indicate that, by depressing the key, the user hasresponded to, and removed, the currently displayed visual stimulus. Suchanimation clues help subjects maintain the appropriate rhythm inresponding to visually and audibly presented stimuli, and preventinadvertent and unwanted latencies that may otherwise arise when asubject becomes confused as to what effect is elicited by input to theinput keys during administration of an SAITA block.

FIGS. 9A-B illustrate an additional aspect of the SAITA directed topreschool children. As shown in FIG. 9A, when a subject incorrectlyresponds to a visual stimulus, such as depressing the right-hand inputkey, currently associated with the insects when presented with theflower visual stimulus 902, the standard ITA displays a red “X” symbol904 to indicate an incorrect response. While acceptable to adultsubjects, such incorrect-response feedback may be disconcerting ortroubling to preschool subjects. As shown in FIG. 9B, in the SAITAdirected to preschool children, rather than displaying a red “X” symbol,as displayed by the standard ITA, a wiggling question mark 906 isdisplayed, and the computer proctor, or a human proctor, explains with aphrase such as: “The computer did not understand. Please try again.”Such neutral feedback maintains a positive attitude of the subjectwhich, in turn, facilitates subject tolerance for continuedstimulus/response interactions.

For the SAITA directed to preschool children, stimuli directed to thefirst conceptual dimension, such as the target concepts {flowers,insects} in the exemplary test of FIGS. 1A-F and 5-9B, are alwayspresented visually, as displayed pictures, while stimuli directed to thesecond conceptual dimension, such as the attribute concepts {pleasant,unpleasant}, are presented in an audio medium, such as by recordedenunciation of a word or phrase. In this way, the key assignments arereinforced and simplified, so that the test subject can more easilymentally categorize presented stimuli. In alternative types of SAITA,alternative types of mappings between stimulus categories and stimuluspresentations may be used to facilitate mental categorization.

FIG. 10 illustrates an additional aspect of the SAITA directed topreschool children. For preschool children, positive feedback iscritical in maintaining a continued engagement of the subject with theSAITA. A positive-feedback display 1002 is therefore displayed atvarious points during the SAITA to encourage the test subject tocontinue with the SAITA and to facilitate a continued positive attitudeof the subject towards the SAITA, despite problems that the subject mayexperience during administration of the SAITA. For example, a positivefeedback screen may be displayed at intervals during key-associationtraining and during each block of the SAITA. In addition, when subjectinattention or fatigue is detected, either by a human test administratoror automatically by SAITA routines running on the computer, a positivefeedback screen may be provided or the subject may be invited to spendtime in a play area both as a mental break and to encourage the subjectto remain attentive during the SAITA.

FIG. 11 illustrates the standard ITA as a control-flow diagram. The

ITA is introduced and explained, and the routines that implement thestandard ITA are configured for the congruent block, in a first step1102. Then, each of steps 1104-1109 are carried out for each block ofthe standard ITA, generally for two blocks representing the congruentand incongruent cases for a particular concept-association hypothesis.In step 1104, pictorial information is provided visually to the subjectto establish concept/response-key combinations used in the firstconceptual dimension. In step 1105, a series of stimulus/responseinteractions are administered to practice thetarget-concept/response-key combinations. In steps 1106 and 1107, theattribute-concept/response-key combinations are established, andstimulus/response interactions are then administered to practice thetarget-concept/response-key combinations. In step 1108, a practice ITAsub-block is administered, as a series of stimulus/responseinteractions, assuming the concept-pair/response-key combinationsestablished in steps 1104-1107. Then an actual ITA sub-block, comprisinga series of stimulus/response interactions, is administered in step1109. After the congruent case is administered, thetarget-concept/response-key combinations orattribute-concept/response-key combinations are internally reversed andthe routines that implement the standard ITA configured for theincongruent case, in step 1112. The standard ITA generally employs afixed number of stimulus/response interactions forconcept-pair/response-key combination reinforcement, each practicesub-block, and the test sub-block. The standard ITA, as discussed above,generally employs only visual stimulus presentation and generally doesnot monitor latencies during test administration to detect problems,including lagging attention, misunderstood instructions, or otherproblems that may arise.

FIGS. 12-17 illustrate a general SAITA using the control-flow-diagramillustration method employed in FIG. 11 to illustrate the standard ITA.It should be noted that many steps can be carried out either by a humantest administrator, by software routines that automate portions of testadministration, of by both a human test administrator and softwareroutines. There are many alternative approaches and examples. Forexample, particular tasks and sub-tasks may be carried out at differentlevels in different approaches and examples. Monitoring for subjectfatigue, for example, may be carried out at the sub-block level, withineach block of an SAITA, or may alternatively be carried out at the blocklevel. Thus, the described example represents only one approach, andmany alternative approaches are possible in which the level at whichtasks are carried out and features provided is altered, certain tasksand/or features are omitted, additional tasks and/or features are added.

In step 1202, a subject-appropriate introduction to the test isprovided. Subject-appropriate introduction may include certain types ofpositive stimulus, to engage the subject. For example, visual displaysmay employ certain types of colors that are found to be conducive toattracting the subject's attention and encouraging cooperation. Audiostimuli, including attractive music or other attention-attracting andpositive-response-inducing stimuli may be presented. For certaincategories of subjects, human administration may be important, ratherthan relying solely on automated administration of the SAITA through acomputer interface. For the SAITA directed to preschool children, ahuman administrator is always present, to add explanations, monitor thetest subject's attitude, performance, and understanding of testprocedures, and to provide positive feedback. Test goals, general testprocedures, and other aspects of the SAITA may be described insubject-appropriate manners, using examples, words, images, and otherpresentations of information appropriate to the category of subject. Instep 1204, the internal routines are configured for the congruent case.In step 1206, a subject-appropriateconcept-pair/response-key-combination training with monitoring isinvoked in order to establish the concept-pair/response-key-combinationsused in the current block of the test. If problems are detected, asdetermined in step 1208, then various ameliorative operations may becarried out in step 1210. For example, additional instruction may benecessary, or a break with toys and other recreational activities may beneeded, in the case of preschool children. In the case of disabledsubjects, attention may need to be paid to various input procedures andinformation displays to ensure that the disabled subject disabledsubject can properly receive and respond to presented information andstimuli. Next, in step 1212, a subject-appropriate test sub-block withmonitoring is administered. After administration of the test sub-block,any problems that are detected, in step 1214, may be ameliorated in anadditional amelioration step 1216. The internal routines arereconfigured for the incongruent case, in step 1218, andconcept-pair/response-key-combination training and test administrationare carried out for the incongruent block by repeating steps 1206, 1208,1210, 1212, 1214, and 1216. Finally, results are computed and feedbackis provided, in step 1220.

FIG. 13 illustrates the subject-appropriate concept-pair/response-keycombinations training with monitoring routine called in step 1206 ofFIG. 12. In step 1302, a concept/response routine is called with respectto the target-concept/response-key combinations used in the currentblock of the test. If any problems are detected in step 1304, thetarget-concept/response-key-combination training may be repeated, asdetermined in step 1306, or problems may be ameliorated by additionalprocedures, in step 1308. When problems cannot be detected at thecurrent level, a problem indication is returned in step 1310. Similarly,the concept/response routine is called, in step 1312, to establishattribute-concept/response-key combinations used in the current block ofthe test. Again, when problems are detected, in step 1314, theattribute-concept/response-key-combination training is either repeatedor problems are ameliorated, in step 1316. When theconcept-pair/response-key training is completed with all the problemsaddressed, and no problems remaining, then the routine returns with a“no problem” indication in step 1318.

FIG. 14 illustrates the concept/response routine called in steps 1302and 1312 of FIG. 13. In step 1402, concept-pair/response-key training ispresented in a subject-appropriate manner. For example, in an SAITAdirected to preschool children, the concept-pair/response-key trainingmay be presented using color-associated collages of pictures, asdescribed above with reference to FIG. 5. Alternatively,concept-pair/response-key training may be described by a humanadministrator or by audio statements with visual indications displayedon the computer display. In step 1404, either a human administrator orthe automated system queries the subject to determine whether thesubject has understood the presented concept-pair/response-keycombinations, in step 1402. When the subject matter has not beenunderstood, as determined in step 1406, then the presentation may berepeated or, when already repeated, an error may be returned in step1410. When the concept-pair/response-key combinations have beenunderstood, then the internal routines set up a running average andrunning variance computation, in step 1412. Then, in a for-loopcomprising steps 1414-1419, a number of stimulus/response trials arecarried out to allow the subject to practice using theconcept-pair/response-key combinations. Note that the number of trials nis selected to be appropriate for the category of subjects being tested.In step 1415, the routine “next trial” is invoked in order to carry outthe next stimulus/response interaction. If problems are detected, asdetermined in step 1416, then the trial may be repeated, as determinedin step 1418, or a problem indication may be returned in step 1420.

FIG. 15 illustrates the routine “next trial” called in step 1415 in FIG.14 and also called in step 1705 of FIG. 17. In step 1502, a timer isset. Then, in step 1504, the stimulus for the trial is presented to thesubject in a subject-appropriate presentation. For example, as discussedabove, in a preschool SAITA, a visually presented stimulus may beaccompanied by an audio tone, and an audio-presented stimulus may beaccompanied by a visually presented word or phrase. Then, in step 1506,the routine waits for a response from the subject or for timerexpiration. If the timer has expired, as determined in step 1508, thenthe subject has failed to respond to the stimulus in a reasonable amountof time. In that case, various operations arc undertaken to handle theproblem, in step 1510 and either an indication that the step should berepeated is returned in step 1512 or a problem indication is returned instep 1514. Timer expiration is tailored to the category of subjects towhich an SAITA is directed. A much larger response window may be allowedfor preschool children, for example, than for adults, reflecting thelonger latencies expected for preschool children. When the timer hasn'texpired, the routine next determines, in step 1516, whether the responseis correct or incorrect. If the response is incorrect, then if theresponse is the first incorrect response, as determined in step 1518,then a subject-appropriate incorrect-response message is presented, instep 1520, either by the computer or by a human test administrator.Control then flows back to step 1506 to wait for the subject to properlyrespond to the stimulus. If the subject has incorrectly respondedpreviously, then the problem is attempted to be ameliorated in step1522. Either an indication to repeat the step is returned, in step 1524,or a problem indication is returned in step 1526. When the response iscorrect, the running variance and latency for the response are computedin step 1528. If either the currently computed variance for responses orthe latency for the current response indicates a problem, as determinedin step 1530, then problem amelioration may he carried out in step 1510.When no problems have occurred, or all problems handled, then a “noproblem” indication is returned in step 1532. When the computed runningvariance passes a threshold value, the subject may have tired or lostinterest in the test, requiring the administrator to provide the subjectwith a break, again explain the purpose of the test, or carry out someother such ameliorative procedure. In certain cases, it may be prudentto discontinue the test due to inability of the subject to continue.

FIG. 16 shows a control-flow representation of the routine“subject-appropriate test with monitoring” called in step 1212 of FIG.12. If a practice test is appropriate for the subject, as determined instep 1602, then the routine “subject-appropriate test” is called in step1604 to administer the practice test. Otherwise, the routine“subject-appropriate test” is called in step 1606 to administer theactual test. In one SAITA developed for preschool children, the practicetest sub-block and test sub-block are merged, or the distinction betweenpractice session and actual test are blurred, to avoid confusingsubjects.

FIG. 17 shows an illustration of the routine “subject-appropriate test”called in steps 1604 and 1606 of FIG. 16. First, a running average andrunning variance computation are initialized, in step 1702. Then, in afor-loop comprising steps 1704-1709, each stimulus/response interactionof a subject-appropriate number n of stimulus/response interactions iscarried out. Any detected problems, in step 1706, are handled by eitherrepeating the trial, as determined in step 1709, or returning a problemindication in step 1712. Finally, in step 1714, results for the test arecomputed and stored, to facilitate final result computation and feedbackprovision in step 1220 in FIG. 12. The number of stimulus/responseinteractions is selected to be appropriate for the category of testsubjects to which the test is directed. While, for example, adults maybe administered 20 stimulus/response interactions, a preschool child,for example, may need to complete 30 or more stimulus/responseinteractions, for example.

In developing SAITAs, collected information may be employed in order todesign appropriate presentation and administration of the SAITA for aparticular category of subjects. FIG. 18 illustrates various types ofinformation that may be employed to facilitate development of SAITAs.For example, a subject table 1802 may associate each category of subjectwith a subject ID. Subject IDs may be then used to indicate thoseaspects and features appropriate for that category of subject in variousadditional tables 1804-1806, and other tables not shown in FIG. 18. Forexample, subject-appropriate colors may be listed in a colors table1804. As another example, subject-appropriate presentation features maybe listed in a presentation-features table 1805. Presentation featuresmay include types of stimulus presentation, visual and audio cuesemployed during tutorials and stimulus/response interactions, and othersuch features. Various tables, such as table 1806, may act as filtersduring design of tests. Table 1806 includes words that are inappropriatefor particular categories of subjects. Filters may be established forvisual images, audio signals, and colors. Of course, far more complexdatabase schemas may be designed for efficiently storingsubject-appropriate parameters, features, and filters. As one example, asubject may be initially interviewed, and pictures of the subject andthe subject's possessions or commonly used items taken, for subsequentuse as stimuli during a later-administered ITA.

FIG. 19 illustrates, in control-flow fashion, the design and methodologyfor designing SAITAs. In step 1902, various types of presentationfeatures, test-flow parameter values, such as the number ofstimulus/response interactions in each sub-block and tutorial, areselected, and filters are employed to detect and substitutesubject-appropriate words, images, sounds, and other presentationfeatures for words, sounds, and other presentation features deemedinappropriate for the category of subject for which the test has beendesigned. Next, in step 1904, an initially designed test is administeredto a number of subjects. In step 1906, the results of administration ofthe initial test are compiled and analyzed. When it is determined thatparticular stimuli in the test have high response latencies or elicit alarge number of incorrect responses or produce a pattern of resultsopposite to what theory would predict (e.g. a whole sample of boysassociating SELF with GIRL), as determined in step 1908, then thestimuli may be replaced with different stimuli and the stimuli that werereplaced may be entered into various filters, in step 1910 (e.g.,replacing the label and/or item SELF with the label and/or item ME).Similarly, if general latency problems, high variance, or othergeneralized problems arc detected for the test, in step 1912, then thepresentation may be altered, and altered presentation features may beadded to filters or various presentation-feature tables may be updatedfor the particular category of subject for which the test is directed.If the number of problems detected after administering the test to anumber of subjects, in step 1904, is greater than a threshold number ofproblems, as determined in step 1916, then the modified test is againadministered to a number of subjects, beginning at step 1904, withcontrol again flowing to step 1906 and beyond in order to again evaluateand modify the test.

Often, when developing and evaluating SAITA tests, it is very useful toexplicitly question a subject about his or her attitudes or feelingstoward the concepts and categories, the strength of association towardswhich are subsequently measured by the SAITA, in order to have explicitresponses to compare to the strengths of associations measured by theSAITA. Such explicit information, derived from Likert-type tests, isused to evaluate scoring methods for ITA tests, as mentioned above. Asone example, FIGS. 20A-F illustrate a Likert-type testing method adaptedfor preschool children, an example of a subject-appropriate Likert-typetesting method (“SALTM”), that solicits explicit responses from subjectsrelated to gender and math skills. The exemplary SALTM shown in FIGS.20A-F comprises 6 panels displayed on a computer screen, such as panel2002 in FIG. 20A, each panel comprising two images, such as images2004-2005 in FIG. 20A. Each image is associated with two input features,such as input features 2006 and 2008 associated with image 2004 in FIG.20A. Selection of the larger input feature 2006 indicates 2006 indicatesstrong agreement by a subject with a proposition posed by the testadministrator related to the image 2002, and selection of the smallerinput feature 2008 indicates less strong agreement by the subject withthe proposition. In general, for each displayed panel, the subject isasked to select one image from the two images that bests answers orrelates to a proposition, and then selects the appropriate input featureto indicate the strength of agreement by the subject that the selectedimage answers or relates to the proposition.

The panels shown in FIGS. 20A-F comprise two three-panel tests, directedto a female subject, the panels of which are directed to: (1) perceivedgender identity; (2) perceived gender stereotype; and (3) perceivedmathself-concept. For example, after display of the first panel, shownin FIG. 20A, the two children in the two images are described by name,e.g. Michael and Emily, and the subject is then asked which image ismost like the subject, to assess the subject's perceived genderidentity. Then, the subject is asked to indicate to what degree theselected images represent the subject. After display of the first panel,shown in FIG. 20B, the characters in the two images are described byname, e.g., Jacob and Sarah, as working on a math problem, and thesubject is then asked which character more enjoys working on the mathproblem, to assess the subject's degree of gender stereotyping withrespect to math ability. After display of the third panel, shown in FIG.20C, the characters in the two images are described by name, e.g.,Jessica and Ashley, as reading and working on a math problem,respectively, and the subject is then asked which character is mostsimilar to the subject, to assess the subject's perceived mathself-concept. The next three panels, shown in Figures D-F, are directedto assessing the same concepts, but change the order of, or activitiesof, the characters in the images. The subject's responses can be used todetermine the subject's explicitly indicated strength of associationbetween gender and math ability, to facilitate evaluation of theeffectiveness of a subsequently administered SAITA test for strength ofassociation between gender and math ability.

FIG. 21 illustrates a general-purpose computer architecture suitable forexecution of IAT, SAITA, and other types of tests. The internalcomponents of many small, mid-sized, and large computer systems as wellas specialized processor-based storage systems can be described withrespect to this generalized architecture, although each particularsystem may feature many additional components, subsystems, and similar,parallel systems with architectures similar to this generalizedarchitecture. The computer system contains one or multiple centralprocessing units (“CPUs”) 2102-2105, one or more electronic memories2108 interconnected with the CPUs by a CPU/memory-subsystem bus 2110 ormultiple busses, a first bridge 2112 that interconnects theCPU/memory-subsystem bus 2110 with additional busses 2114 and 2116, orother types of high-speed interconnection media, including multiple,high-speed serial interconnects. These busses or serialinterconnections, in turn, connect the CPUs and memory with specializedprocessors, such as a graphics processor 2118, and with one or moreadditional bridges 2120, which arc interconnected with high-speed seriallinks or with multiple controllers 2122-2127, such as controller 2127,that provide access to various different types of mass-storage devices2128, electronic displays, input devices, and other such components,subcomponents, and computational resources. The electronic displays,including visual display screen, audio speakers, and other outputinterfaces, and the input devices, including mice, keyboards,touchscreens, and other such input interfaces, together constitute inputand output interfaces that allow the computer system to interact withhuman users.

The magnetic disk platters, optical disks, or solid-state electronicmemories of mass-storage devices are one example of a computer-readablemedium on or within which data can be stored and from which stored datacan be retrieved by a computer system. Additional examples ofcomputer-readable media include removable disks, including optical andmagnetic disks, electronic random access memories of various types, andflash memory. As would be well understood by those familiar withcomputing and electronics, the phrase “computer-readable medium,” in thecontexts of data storage and encoding of instructions of computerprograms, refers to tangible, physical media, such as those describedabove, and not to electromagnetic waves and other transmission mediaused for transmission of data.

Although the present invention has been described in terms of particularembodiments, it is not intended that the invention be limited to theseembodiments. Modifications within the spirit of the invention will beapparent to those skilled in the art. For example, any number ofdifferent implementations of SAITA tests using various differentprogramming languages, computer platforms, and varying differentprogramming parameters, including control structures, variables, modularorganization, and other such parameters are possible. A large variety ofdifferent aspects of ITA development and administration may be varied,systematically, in order to adjust an SAITA optimally to any particularcategory of subjects. In the above description, color, words andphrasing, visual, audio, and other presentation parameters are adjusted,but additional parameters may be adjusted in alternative embodiments ofthe present invention. Stimulus and information presentation media mayinclude visual and audio media, as discussed above, and may additionallyinclude mechanical, electrical, and other types of presentation mediaperceptible to subjects.

It is appreciated that the previous description is provided to enableany person skilled in the art to make or use the present disclosure.Various modifications to the disclosed examples of SAITAs will bereadily apparent to those skilled in the art, and the generic principlesdefined herein may be applied to other types of SAITAs and other testswithout departing from the spirit or scope of the disclosure. Thus, thepresent disclosure is not intended to be limited to that disclosed inthis document, but is instead to be accorded the widest scope consistentwith the principles and novel features disclosed herein.

The invention claimed is:
 1. A storage from which instructions areloaded into memory, the storage having stored thereoncomputer-executable instructions for implementing one or more routinesthat together compose a subject-appropriate implicit test ofassociations, the subject-appropriate implicit test of associationscomprising: a subject-appropriate introduction that presents informationto a test subject through one or more computer-system output interfaces;and two or more blocks, each block comprising— a subject-appropriatetutorial sub-block configured to introduce to a test subject, and trainthe test subject that— when stimuli corresponding to either of a firstpair of concepts is displayed, the test subject should enter a firstresponse through a first input feature; and when stimuli correspondingto either of a second pair of concepts is displayed, the test subjectshould enter a second response through a second input feature, whereinthe subject-appropriate tutorial sub-block is further configured tomonitor and detect misunderstandings and problems in a first performanceof the test subject; and a subject-appropriate test sub-block configuredto— administer a subject-appropriate number of interactions, whereineach interaction comprises a subject-appropriate pairing of astimulus-presentation with a test-subject-response; and monitor anddetect misunderstandings and problems in a second performance of thetest subject; wherein at least one of the subject-appropriateintroduction, subject-appropriate tutorial sub-block, andsubject-appropriate test sub-block was created by— determining a testsubject category; and matching to the test subject category—one or moreparticular computer-system input configurations or one or moreparticular computer-system output configurations, and wherein the testsubject is associated with the test subject category.
 2. The storage ofclaim 1 wherein the subject category is preschool children.
 3. Thestorage of claim 2 wherein: the first concept of the first pair ofconcepts and the first concept of the second pair of concepts togethercomprise target concepts; the target concepts form a first conceptualdimension; and each target concept is assigned, by thesubject-appropriate tutorial sub-block, to one of the first inputfeature or the second input feature; and wherein— the second concept ofthe first pair of concepts and the second concept of the second pair ofconcepts together comprise attribute concepts; the attribute conceptsform a second conceptual dimension, and each attribute concept isassigned, by the subject-appropriate tutorial sub-block, to a differentone of the two input features.
 4. The storage of claim 3 wherein stimuliand information are presented to the test subject using one or morepresentation media selected from: visual presentation using graphicsand/or images displayed on a computer display screen; audio presentationusing recorded spoken passages, musical tones, and/or other recorded orsynthesized sounds through computer speakers; mechanical presentationvia devices employed by the blind and disabled; and electricalpresentation via devices employed by the blind and disabled.
 5. Thestorage of claim 4 wherein stimuli directed to the first conceptualdimension are presented, during administration of thesubject-appropriate number of interactions, using a first presentationmedium and wherein stimuli directed to the second conceptual dimensionare presented, during administration of the subject-appropriate numberof interactions, using a second presentation medium different from thefirst presentation medium.
 6. The storage of claim 5 wherein: stimulidirected to at least one of the first conceptual dimension or the secondconceptual dimension are presented visually; the graphics or imagesdirected to each concept of the conceptual dimension are displayed abovecolor-coded regions of the computer display screen as collages; and eachcolor-coded region matches the color of the input feature to which aresponse is input when the graphics or images are displayed.
 7. Thestorage of claim 5 wherein the stimuli directed to the first conceptualdimension are presented using the first presentation medium along withcues presented using the second presentation medium; and wherein thestimuli directed to the first conceptual dimension are presented usingthe second presentation medium along with cues presented using the firstpresentation medium.
 8. The storage of claim 5 wherein the asubject-appropriate tutorial sub-block and the subject-appropriate testsub-block are further configured to provide, at intervals, positivefeedback presentations.
 9. The storage of claim 5 wherein thesubject-appropriate test sub-block is further configured to, when asubject incorrectly responds to a presented stimulus, provide neutral,subject-appropriate feedback.
 10. The storage of claim 1 wherein thesubject-appropriate tutorial sub-block or the a subject-appropriate testsub-block monitors and detects misunderstandings by: computing theelapsed time between presentation of a stimulus and detection of aresponse by a test subject; maintaining a running variance and latencycalculation and updating the running variance and latency calculationafter computing the elapsed time between presentation of a stimulus anddetection of a response by a test subject; and determining whether thecomputed elapsed time is greater than a latency threshold value andwhether the running variance is greater than a variance threshold.
 11. Amethod for administering a subject-appropriate implicit test ofassociations through one or more computer-system input and outputinterfaces, the method comprising: providing, by a computer-systemoutput interface, a subject-appropriate introduction that presentsinformation to a test subject through the one or more computer-systemoutput interfaces; and administering two or more blocks, the two or moreblocks comprising— a subject-appropriate tutorial sub-block configuredto, using the computer-system output interface, introduce to a testsubject and train the test subject that— when stimuli corresponding toeither of a first pair of concepts is displayed, the test subject shouldenter a first response through a first input feature; and when stimulicorresponding to either of a second pair of concepts is displayed, thetest subject should enter a second response through a second inputfeature, wherein the subject-appropriate tutorial sub-block is furtherconfigured to monitor and detect misunderstandings and problems in afirst performance of the test subject; and a subject-appropriate testsub-block configured to— administer a subject-appropriate number ofinteractions, wherein each interaction comprises a subject-appropriatepairing of a stimulus-presentation with a test-subject-response; andmonitor and detect misunderstandings and problems in a secondperformance of the test subject; wherein at least one of thesubject-appropriate introduction, subject-appropriate tutorialsub-block, and subject-appropriate test sub-block was created by—determining a test subject category; and matching to the test subjectcategory—one or more particular computer-system input configurations orone more particular computer-system output configurations, wherein thetest subject is associated with the test subject category.
 12. Themethod of claim 11 wherein the subject category is preschool children.13. The method of claim 12 wherein: the first concept of the first pairof concepts and the first concept of the second pair of conceptstogether comprise target concepts; the target concepts form a firstconceptual dimension; and each target concept is assigned, by thesubject-appropriate tutorial sub-block, to one of the first inputfeature or the second input feature; and wherein— the second concept ofthe first pair of concepts and the second concept of the second pair ofconcepts together comprise attribute concepts; the attribute conceptsform a second conceptual dimension; and each attribute concept isassigned, by the subject-appropriate tutorial sub-block, to a differentone of the two input features.
 14. The method of claim 13, furthercomprising presenting stimuli and information to the test subject usingone or more presentation media selected from: visual presentation usinggraphics and/or images; audio presentation using recorded spokenpassages, musical tones, and/or other recorded or synthesized sounds;mechanical presentation via devices employed by the blind and disabled;and electrical presentation via devices employed by the blind anddisabled.
 15. The method of claim 14, further comprising presentingstimuli directed to the first conceptual dimension, duringadministration of the subject-appropriate number of interactions, usinga first presentation medium and presenting stimuli directed to thesecond conceptual dimension, during administration of thesubject-appropriate number of interactions, using a second presentationmedium different from the first presentation medium.
 16. The method ofclaim 15, further comprising: visually presenting stimuli directed toone conceptual dimension; displaying the graphics or images directed toeach concept of the conceptual dimension above color-coded regions ofthe computer display screen as collages, with each color-coded regionmatching the color of the input feature to which a response is inputwhen the graphics or images are displayed.
 17. The method of claim 15,further comprising: presenting the stimuli directed to the firstconceptual dimension using the first presentation medium along with cuespresented using the second presentation medium; and presenting thestimuli directed to the first conceptual dimension using the secondpresentation medium along with cues presented using the firstpresentation medium.
 18. The method of claim 15, further comprisingproviding, by the subject-appropriate tutorial sub-block or by thesubject-appropriate test sub-block, at intervals and when monitoringdetects problems or lack of attention on the part of the test subject,positive feedback presentations.
 19. The method of claim 15, furthercomprising, when a subject incorrectly responds to a presented stimulus,providing neutral, subject-appropriate feedback.
 20. The method of claim11, further comprising administering, by a human administrator, at leastone of the two or more blocks.
 21. The method of claim 11 wherein thesubject-appropriate tutorial sub-block or the a subject-appropriate testsub-block monitors and detects misunderstandings by: computing theelapsed time between presentation of a stimulus and detection of aresponse by a test subject; maintaining a running variance and latencycalculation and updating the running variance and latency calculationafter computing the elapsed time between presentation of a stimulus anddetection of a response by a test subject; and determining whether thecomputed elapsed time is greater than a latency threshold value andwhether the running variance is greater than a variance threshold. 22.The method of claim 11, further comprising providing a response inputdevice with only two input features, the two input features comprising afirst input feature and a second input feature, wherein the first inputfeature has a first color and the second input feature has a secondcolor, and wherein the first input feature changes color to a thirdcolor and the second input feature changes color to a fourth color. 23.A method for developing a subject-appropriate implicit test ofassociations, the method comprising: selecting a category of subjects towhich the subject-appropriate implicit test of associations is directed;selecting, based on the selected category, presentation features;selecting, based on the selected category, a response input deviceappropriate for the selected test target; developing, based on theselected category and using the selected presentation features, blockswith chosen interactions for the test target, wherein a number of thechosen interactions is based on the selected category; substituting, forany words, phrase, audio stimuli, visual stimuli, or colors previouslydetermined to be inappropriate for the selected category, words, phrase,audio stimuli, visual stimuli, or colors that have not been previouslyfound inappropriate for the selected category; providing positivefeedback presentations, wherein a frequency and a content of thepositive feedback presentations are based on the selected category;incorporating monitoring routines and methods to determine whether atest subject remains fully engaged, alert, and positively orientedtowards the test; and employing subject-appropriate Likert-type testingof strengths of association to complement and facilitate evaluation ofstrengths of association measured by the subject-appropriate implicittest of associations.